actually, on further reflection - no, we don't want to break out of the loop just yet. as the comment above says, we want to make sure error channel also gets processed, which is exactly why we replace outCh with a new one and wait for ss.errCh to be closed as well.

Ah wasn't sure what you want to break out of, I guess you meant the select then?
Would continue loop be clearer in that case?

great idea, indeed that would make the intent more clear. done.

Is this the channel used to communicate between stepRunner and targetHandler. Could you please clarify in the comment? I haven't read through the whole code yet and it's not clear.

Does waitStepRunners guarantee that all steps have returned? If so, stepCancel should not be necessary. If not, steps should be awaited for if a cancellation signal is being propagated. We should keep track of any step that does not return within the timeout and flag it accordingly.

you are right, this is not needed as by this point all the step runners have either exited or we gave up waiting on them to exit.

If there is even only one target that failed in any step, we are declaring the test not resumable (but continuing the loop after we declare resumeOk false)? It should be fine to fail a target in a step, and still be able to resume the test.

stepState.runErr aka stepErr is not about target errors, it is

// Runner error, returned from Run() or an error condition detected by the reader.

i.e. it's an error condition that aborts entire run. presence of such errors making run non-resumable makes sense.

Well, ok, not sure why I missed that here we were referring to an error of the whole step.

This is an example of the indirect data path I was mentioning before.

Intuitively, I would imagine that tr.steps is protected by tr.mu. Indeed, this goroutine acquires mu throughout its whole execution and I see we read ss.<X> and then waits for tr.cond.Wait. Is somebody else, from outside, changing stepRunning, readerRunning, etc. while we hold mu?

From looking at the code, yes, but those writers also hold mu. So, I cannot wrap my head around how this can work.

this has to do with how waiting on sync.Condition works. when you want to wait for some internal state protected by a mutex to change (in this case stepRunning and readerRunning), you acquire a mutex, test the state and if it is not to your satisfaction, you invoke tr.cond.Wait() which releases the mutex and then suspends until signaled, at which point it re-locks and returns, letting you re-examine the state again. we signal tr.cond when things change (runner exits, rader exits of target handler exits).

Previous implementation here would also issue a termination signal to the whole test in a further attempt to make the pipeline shut down.

the step runner context will eventually be canceled (via defer stepCancel())

Looks like safeCloseOutCh and safeCloseErrCh can be merged together and accept an argument from outside.

yes, except channels have different types and i couldn't find a way to cast them to something common.

If outCh indicates that the test step is guaranteed to have returned, then whoever signaled completion by closing outCh should have closed also errCh. Responsibility of closing these channels should not be spread across multiple entities.

this will be simplified when we have one channel for results (as discussed). i'll leave it as is for now.

This should not be necessary. It should be stepRunner guaranteeing to us that step termination is protected against shutdownTimeout. We shouldn't be doing it in multiple places.